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(57) Abstract: Watermarks arc added to data 
streams (e.g.» video streams) to enable improved 
detection and thereby inevention of the tfaeft of 
such data streams. In one embodiment, different 
watennarks are added at two or more different 
stages along a primaiy distribution path in order 
to determine at what location a particular copy 
of the video stream diverged, if at all, firom 
the primary distribution path prior to the end 
of the path. In this way, a particular copy of a 
video stream can be analyzed to determine who 
was the intended end user of the video stream. 
These features can be implemented to aid in the 
detection of content tfaeft and, thiou£^ related 
deterrent effects, aid in die prevention of such 
tfaeft in the first place. 
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WATERMARKING DATA STREAMS AT MULTIPLE DISTRIBUTION STAGES 
BACKGROUND OF THE INVENTION 

Field of the Invention 

5 The invention relates to data processing and, more particularly, to digital wateimarking of data 

streams, such as digital image sequences. 

Cross-Reference to Related Applications 

Tlie subject matter of this application is related to U.S. patent application no. 09/001,205 Cthe 
10 '20S application**) filed 12/30/97 as attorney docket no. 12602, the teachings of which are incorporated 
herein by reference. The *205 application describes techniques for mserting a watermark into a 
compressed video bitstream without first having to fully decode the bitstream. 

Description of the Related Art 

1 S With the advent of digitization of images, digital image distribution, and digital video 

availability, copyright protection of such digital imageiy has become a substantial issue for image 
publishers and authors. One technique used to identify digital video ownership is to embed into an 
image sequence a digital '"watermaric" containing information identifying the owner. Such watermarks 
should (a) be secure and robust to intentional corruption and to compression processing, (b) not be 

20 unreasonably complex to embed and extract, and (c) be compatible and interoperable with conventional 
image processing systems. For authentication applications, the watermark is generally invisible to a 
viewer of a decoded image sequence. However, in some applications, it is desirable to produce a 
visible watermark that can be removed by an authorized image decoder, but cannot be removed by an 
unauthorized decoder. 

25 Illegal copying of video content is a serious problem. Such copying may occur at any of 

various different stages along the distribution path of video content as it is transmitted from the original 
producer of the video content to the fuial end user (i.e., the viewer) of that content. 

SUMMARY OF THE INVENTION 
30 The present invention is directed to techniques for using digital watermarking to improve the 

ability to detect where along the distribution path from producer to viewer illegally obtained video 
content was pirated. As such, the present invention may provide a deterrent effect that will aid in the 
prevention of such content theft in the first place. 
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According to one embodiment, the present invention is a method for processing a data stream, 
comprising the steps of adding a plurality of different watermarks to the data stream at a plurality of 
different stages along a primary distribution path for the data stream thereby enabling detection of a 
location along the primaiy distribution path of divergence of the data stream fiom the primaiy 
distribution path by analyzing the watennarics in the data stream. 

According to an alternative embodiment, the present invention is a method for processing a 
data stream, comprising the steps of (a) analyzing the data stream to identify a set of one or more 
different watennarks contained in the data stream, wherem each different watermark corresponds to a 
different stage along a primary distribution path for the data stream; and (b) determining where the data 
stream diverged from the primary distribution path based on the identified set of watennarics. 

BRIEF DESCR IPTION OF THE DRAWINGS 
Other aspects, features, and advantages of the present invention will become more fully 
apparent fiom the following detailed description, the appended claims, and the accompanying drawings 
in which Fig. 1 shows a set of four different distribution paths for video streams, according to one 
embodiment of the present invention. 

DETAILED DESCRTPTTON 

F^g. I shows a set of four different distribution paths for video streams, according to one 
embodiment of the present invention. Each distribution path begins with the same three stages: video 
source 102, encoder 104, and storage device 106. Video source 102, which may be a digital video 
camera or other suitable device, generates the original video stream in a pixel domain. Encoder 104 
applies a suitable digital video compression algorithm to generate a compressed video bitstream for the 
video stream, which bitstream is then stored on storage device 106. At appropriate times, the 
compressed video bitstream is read from storage device 106 for transmission by an appropriate 
transmission device 108 to a decoder 110, where the compressed video bitstream is decoded back into a 
pixel domain for display on a suitable display device 112 (e.g., a television set or computer monitor). 

As shown in Fig. 1, at each stage along the four different distribution paths, a different 
watennark (i.e., one of watennarics WM#1 through WM#13) is added to the video stream. In general, a 
watermark may contain any useful infonnation, but will preferably contain at least (1) a stage ID that 
identifies the stage at which the watermark was added and (2) a time tag that identifies the time and 
date at which the watermark was added to the video stream. 
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Video source 102, encoder 104, each decoder 110, and each display device 112 all preferably 
add their corresponding watermarks into the video stream in the pixel domain, while storage device 106 
and each transmission device 108 all preferably add their corresponding watermarks into the 
corresponding compressed video bitstream in the compressed domain, e.g., relying on the teachings in 
5 the '205 patent application. 

After completely traversing one of the four distribution paths in Fig. 1, a particular copy of the 
video stream will have one of the following four sets of watermarks: 

o Distribution Path A consisting of video source 102, encoder 104, storage device 106, 
transmission device #1, decoder #1, and display device #1 : {WM#1, WM#2, WM#3, WM#4, WM#6, 
10 andWM#10}; 

o Distribution Path B consisting of video source 102, encoder 104, storage device 106, 
transmission device #1, decoder #2, and display device #2: {WM#1, WM#2, WM#3, WM#4, WM#7, 
and WM#11}; 

o Distribution Path C consisting of video source 102, encoder 104, storage device 106, 
15 transmission device #2, decoder #3, and display device #3: {WM#1, WM#2, WM#3, WM#5, WM#8, 
andWM#12};and 

o Distribution Path D consisting of video source 102, encoder 104, storage device 106, 
transmission device #2, decoder #4, and display device #4: {WM#1, WM#2, WM#3, WM#5, WM#9, 
andWM#13}. 

20 Each of the four distribution paths shown in Fig. 1 may be referred to as a primary distribution 

path. It may be that the four primary distribution paths shown in Fig. 1 are the only proper (i.e., 
legitimate or authorized) paths for distribution of a particular video stream. A copy of the video stream 
that follows one of these four primary distribution paths from beginning (i.e., video source 102) to end 
(i.e., one of the four display devices 112) will have a corresponding one of the four sets of watermarks 

25 listed above. 

If a copy of the video stream is diverted from one of the four primary distribution paths prior to 
reaching one of the four display devices 112, that copy will not have a complete set of watermarks, and 
the watermarics that are contained in the copy will help identify exactly where (i.e., after what stage) 
along one of the distribution paths the video stream was diverted. 
30 For example, if a particular copy of the video stream contains the set of watermarks consisting 

of { WM#1 . WM#2, WM#3, WM#4} , then it would be known that the video stream diverged from 
either Distribution Path A or Distribution Path B after being processed by transmission device #1 108. 
In this way, the present invention can be used to detect where along the distribution path from producer 
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to viewer illegally obtained video content was pirated. As such, the present invention may also provide 
a deterrent effect that will aid in the prevention of such content theft in the first place. 

In one possible implementation, each transmission device 108 could be a different microwave 
transmitter broadcasting a video stream corresponding to a pay-per-view or other controlled or 
5 regulated broadcast to a different set of authorized viewers. If someone were improperly capturing and 
illegally distributing copies of the pirated video stream, an analysis of one of the pirated copies 
containing the watermarks {WM#1, WM#2, WM#3, WM#4} could at least identify that the theft 
occurred during a particular microwave transmission of the video stream somewhere within the 
coverage area of transmission device #1 at a known date and time. This information may be very 

10 helpful in tracking down the individuals responsible for the content theft and illegal distribution. 

The exact types of watermarks that are added at the various stages may determine whether - 
and what kind of - special equipment is needed to perform the analysis of the video stream to 
determine which watermarks are present. In preferred embodiments, the watermarks are not visible to 
a viewer when the decoded video stream is played back at normal speeds. This can be achieved, for 

1 5 example, by placing watermarks only into a known subset of frames in the video stream (e.g., as few as 
a single frame) and/or by encoding the watermark information so that it is only decipherable by special 
equipment implementmg proprietary decode processing. 

Note that the processing along any one distribution path in Fig. 1 could be continuous in time, 
where all stages process different portions of the video stream at the same time (perhaps bypassing 

20 storage device 106), or discontinuous in time, where at least one stage (e.g., storage device 106) 
completes its processing of the video stream before the next stage (e.g., a transmission device 108) 
begins its processing. The distribution paths shown in Fig. 1 are merely exemplary; it will be 
understood that the present invention can be applied to any suitable distribution path having a plurality 
of different processing stages, including suitable types of processing stages not shown in Fig. 1. For 

25 example, a particular distribution path might include a video retailer who sells or rents video conteiit 
that contains a watermark identifying the video retailer. Such a watermark could be added by the video 
wholesaler who distributes copies of the video content to the video retailer. Ahematively, the 
watermark could be added by the video retailer at the time of sale/lease by the customer, thereby 
adding the ability to include a more relevant time tag into the watermaric, 

30 The present invention is not limited to video streams, but can probably be applied to the 

distribution of any other type of data stream as well, in order to determine the source of illegal copying 
of that data stream. When applied to a video stream, a watermark corresponds to any of the standard 
variety of techniques for encoding watermark information into a video stream, including those 
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techniques in which the watermark is virtually - if not completely - undetectable to viewers during 
normal playback of the video stream. When applied to other types of data streams, such as audio data, 
text data, or any other suitable type of data, a "watermark*' corresponds to information encoded into 
those data stream in a manner analogous to that used for video streams. For example, when applied to 
5 audio streams, a "watermark" may correspond to information encoded into audio stream in a manner 
such that the information is imperceptible to a listener dixring normal audio playback of the audio 
stream. Similarly, when applied to streams of text, a "watermark" may correspond to information 
encoded into the text stream in a manner such that the information does not appear on either (a) the 
graphical display presented to the user or (b) the corresponding printed hardcopy. In all such cases, 

1 0 including that of video watermarking, special processing equipment may be needed to analyze the data 
stream to detect and identify the different watermarks that are in fact present in the stream to determine 
if and where the data stream diverged from a primary distribution path. 

The present invention can be embodied in the form of methods and apparatuses for practicing 
those methods. The present invention can also be embodied in the form of program code embodied in 

1 5 tangible media, such as floppy diskettes, CD-ROMs, hard drives, or any other machine-readable 

storage medium, wherein, when the program code is loaded into and executed by a machine, such as a 
computer, the machine becomes an apparatus for practicing the invention. The present invention can 
also be embodied in the form of program code, for example, whether stored in a storage medium, 
loaded into and/or executed by a machine, or transmitted over some transmission medium or carrier, 

20 such as over electrical wiring or cabling, through fiber optics, or via electromagnetic radiation, 

wherein, when the program code is loaded into and executed by a machine, such as a computer, the 
machine becomes an apparatus for practicing the invention. When implemented on a generaUpurpose 
processor, the program code segments combine with the processor to provide a unique device that 
operates analogously to specific logic circuits. 

25 It will be further understood that various changes in the details, materials, and arrangements of 

the parts which have been described and illustrated in order to explain the nature of this invention may 
be made by those skilled in the art without departing from the principle and scope of the invention as 
expressed in the following claims. 
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CLAIMS 

What is claimed is: 

1 1 . A method for processing a data stream, comprising the steps of adding a plurality of different 

2 watermarks to the data stream at a plurality of different stages along a primary distribution path for the 

3 data stream thereby enabling detection of a location along the primary distribution path of divergence 

4 of the data stream from the primary distribution path by analyzing the watermarks in the data stream. 

1 2. The invention of claim 1, wherein the data stream is a video stream. 

1 3. The invention of claim 2, wherein, along the primary distribution path, at least one watermark 

2 is added to a compressed bitstream for the video stream without having to fully decode the compressed 

3 bitstream. 

1 4. The invention of claim 3, wherein, at two or more different stages along the primary 

2 distribution path, two or more different watermarks are added to the compressed bitstream without 

3 having to fully decode the compressed bitstream. 

1 5 . The invention of claim 4, wherein at least one watermark is added to the video stream in an 

2 original pixel domain near a beginning of the primary distribution path. 

1 6. The invention of claim S, wherein at least one watermaric is added to the video stream in a 

2 decompressed pixel domain near an end of the primary distribution path. 

1 7. The invention of claim 6, wherein each watermark identifies a stage of the primary distribution 

2 path and a time tag corresponding to when processing of the data stream occurred at the corresponding 

3 stage. 

1 8. The invention of claim 3, wherein at least one watermark is added to the video stream in an 

2 original pixel domain near a beginning of the primary distribution path. 

1 9. The invention of claim 8, wherein at least one watermark is added to the video stream in a 

2 decompressed pixel domain near an end of the primary distribution path. 
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1 10. The invention of claim 1, wherein each watermark identifies a stage of the primary distribution 

2 path and a time tag corresponding to when processing of the data stream occurred at the corresponding 

3 stage. 

1 1 1 . A method for processing a data stream, comprising the steps of: 

2 (a) analyzing the data stream to identify a set of one or more different watermarks contained in the 

3 data stream, wherein each different watermark corresponds to a different stage along a primary 

4 distribution path for the data stream; and 

5 (b) determining where the data stream diverged from the primary distribution path based on the 

6 identified set of watermarks. 

1 1 2. The invention of claim 1 1 , wherein the data stream is a video stream. 

1 13. The invention of claim 12, wherein, along the primary distribution path, at least one watermark 

2 was added to a compressed bitstream for the video stream without having to fully decode the 

3 compressed bitstream. 

1 14. The invention of claim 13, wherein, at two or more different stages along the primary 

2 distribution path, two or more different watermarks were added to the compressed bitstream without 

3 having to fully decode the compressed bitstream. 

1 IS. The invention of claim 14, wherein at least one watermark was added to the video stream in an 

2 original pixel domain near a beginning of the primary distribution path. 

1 16. The invention of claim IS, wherein at least one watermark was added to the video stream in a 

2 decompressed pixel domain near an end of the primary distribution path. 

1 17. The invention of claim 16, wherein each watermark identifies a stage of the primary 

2 distribution path and a time tag corresponding to when processing of the data stream occurred at the 

3 corresponding stage. 
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1 1 8. The invention of claim 13, wherein at least one watermark was added to the video stream in an 

2 original pixel domain near a beginning of the primary distribution path. 

1 19. The invention of claim 18, wherein at least one watermaric was added to the video stream in a 

2 decompressed pixel domain near an end of the primary distribution path. 

1 20. The invention of claim 1 1, wherein each watermark identifies a stage of the primary 

2 distribution path and a time tag corresponding to when processing of the data stream occurred at the 

3 corresponding stage. 

1 21 . A machine-readable medium, having encoded thereon program code, wherein, when the 

2 program code is executed by a machine, the machine implements the steps of: 

3 (a) analyzing the data stream to identify a set of one or more different watermarks contained in the 

4 data stream, wherein each different watermark corresponds to a different stage along a primary 

5 distribution path for the data stream; and 

6 (b) determining where the data stream diverged from the primary distribution path based on the 

7 identified set of watermarks. 



4/27/05, EAST Version: 2.0.1.4 



wo 01/13642 



PCTAJSOO/22144 



1/1 



102-2_ 



104. 




COMPRESSED 



106 



108. 




WM«3 



COMPRESSED 



TRANSIilSSION 
DEVICE m 



110 COMPRESSED 

!i 



WUff4 



DOMAIN 



WM#IO 



DEa 


)DER 


PIXEL 


OOHAIN 


DISPLAY 
DEVICE #1 



«M#7 



112 



DEa 
«; 


)DER 
> 


PIXEL 


DOUAIN 


DISPLAY 
DEVICE #2 



WM#8 



TRANSHISSION 
DEVICE «2 ' 



COMPRESSED 



ms 



DOMAIN 



DEa 
«: 


)DER 


PIXEL 


DOMAIN 

r 


DISPLAY 
DEVICE #3 




WM#12 



4/27/05, EAST. Version: 2.0.1.4 



INTERNATIONAL SEARCH REPORT 


Inten ^nal Application No 

PCT/US 00/22144 


A. CLASSIFICATION OF SUBJECT MATTER 

IPC 7 H04N7/24 








According to Intemaiiona) Patent Classificalion (IPC) or to both nationsU ctassiflcatton and IPC 






B. RELDS SEARCHED 


l^inlrTKjm docurneniation seardied (dassitication system foDowed by dassllication synnbols) 


Documentation searched other than minimum documentation to the extent that such documents are included in the fields searched 


Electronic data I>as6 corisulted during the international search (name ot data base and, where practicaL search terms used) 

EPO-Internal , WPI Data, PAJ, INSPEC, COMPENDEX 


C. DOCUMENTS CONSIDERED TO BE RELEVANT 


Category* 


Citation ot document, with indication, where appropriate, of the retevani passages 


Ftelevant todalmNo. 


P.X 


BARNETT R: "Digital watermarking: 
applications, techniques and challenges" 
ELECTRONICS i COMMUNICATION ENGINEERING 
JOURNAL, AUG. 1999, I EE, UK, 
vol . 11, no. 4, 

18 August 1999 (1999-08-18), pages 
173-183, XP002153906 
ISSN: 0954-0695 
abstract 

page 174, column 2, line 32 -page 175, 

column 1, line 4; figure 1 

page 176, line 26 - line 53 

page 179, column 2, line 11 -page 180, 

column 1, line 10 




I- 3. 

II- 13,21 






/~ 






1 y j Further documents are listed In the continuation of tx)x C. 


[y j Patent family members are listed in annex. 


* Special categories of dted documents : 

*A' document defining the general state of the art which Is not 

considered to be of particular relevance 
*E' earlier document but published on or after the international 

filing date 

•L' document which may throw doubts on priority daim(s) or 
which is cited to establish the publication date of another 
citation or other special reason (as specified) 

'QT document referring to an oral disdosure, use. exhibttion or 
other means 

'P* document published prior to the international filing date but 
later than the priority date claimed 


*T* later document published after the international filing date 
or priority date and not in conflid with the application but 
dted to understand the prindple or theory underlying the 
invention 

'X' document of particular relevance; the claimed invention 
cannot be considered novel or cannot te considered to 
involve an inventive step when the docunrtent is taken atone 

*Y* document of particular relevance: the claimed invention 
cannot be considered to involve an inventive step when the 
document is combined with one or more other such docu- 
ments, sud) combinatbn being obvkMis to a person skilted 
in the an. 

'&' document member of the same patent family 


Date of Ihe actual completion of the international search 


Date of mailing of the tntemaitonal search report 


27 November 2000 


08/12/2000 




Name and mailing address of the ISA 

European Patent Office. P.B. 5618 Patentlaan 2 
NL - 2280 HV Rijswijk 
Tel (+31-70) 340-2040. Tx. 31 651 epo nl. 
Fax: (+31-70)340-3016 


Authorized officer 

Beaudet, J-P 



Form PCT/ISA/210 (saeond shast) guV 1992) 

4/27/05, EAST Version: 2.0.1.4^ , 9 



INTERNATIONAL SEARCH REPORT 



Inter. inal Application No 

PCT/US 00/22144 



C.<Contlnuation) DOCUMENTS CONSIDERED TO BE RELEVANT 


Categoiy" 


Citation of document, with Indication.where appropriate, of the relevant passages 


Relevant to daim No. 


X 
Y 


CARONNI G: "ASSURING OWNERSHIP RIGHTS FOR 
DIGITAL IMAGES" 

PROCEEDINGS DER GI-FACHTAGUNG VIS, XX, XX, 
1995, pages 251-263. XP000668970 
abstract 

paragraphs '0001!, '0002!, '03.2! 


I- 3, 

II- 13,21 

4.5.8. 
10,14, 
15,18,20 


X 


ZHAO J: "a www service to embed and prove 
digital copyright watermarks" XP000199921 
abstract 

paragraph '0002! 


1.2.11, 
12,21 


X 


SWANSON M D ET AL: "MULTIRESOLUTION 
SCENE-BASED VIDEO WATERMARKING USING 
PERCEPTUAL MODELS" 
IEEE JOURNAL ON SELECTED AREAS IN 
COMMUNICATIONS, IEEE INC. NEW YORK, US, 
vol. 16, no. 4, 1 May 1998 (1998-05-01), 
pages 540-550, XP000765114 
ISSN: 0733-8716 
paragraph 'OOOC! 


1.2.11. 
12,21 


Y 


HARTUNG F ET AL: "FAST PUBLIC-KEY 
WATERMARKING OF COMPRESSED VIDEO" , SANTA 
BARBARA, CA, OCT. 26-29, 1997, LOS 
ALAMITOS, CA: IEEE, US, PAGE(S) 528-531 
XP000792828 
ISBN: 0-8186-8184-5 
page 528, column 2, line 12 - line 15 
paragraph '02.1! 


4,5,8, 
14,15.18 


Y 
A 


■ US 5 739 864 A (COPELAND GREGORY C) 
14 April 1998 (1998-04-14) 
claim 2 


10,20 
7.17 


A. 


HARTUNG AND GIROD: "DIGITAL WATERMARKING 

OF RAW AND COMPRESSED VIDEO" 

PROCEEDINGS OF THE SPIE.SPIE, BELLINGHAM, 

VA,US, 

vol. 2952, 7 October 1996 (1996-10-07), 

pages 205-213. XP002085796 

abstract 

paragraph '0001!; figure 2 
paragraph '0004! 


3 


A 


WO 99 29114 A (AT & T CORP) 
10 June 1999 (1999-06-10) 
abstract 

page 2, line 10 - line 12 
page 6, line 10 - line 19 
page 9, line 15 -page 10, line 14 


8.18 



Fbfm PCT/ISA/210 (continuation of seoond she 



4/27/05, EAST Version: 2.0.1.4 



INTERNATIONAL SEARCH REPORT 

Information on patent family members 



Inten nal Application No 

PCT/US 00/22144 



Patent document 
dted in search report 



Publication 
date 



Patent family 
member(s) 



Publication 
date 



698870 B 


12- 


-11- 


-1998 


3370395 A 


14- 


-03- 


-1996 


9508624 A 


18- 


-11- 


-1997 




90- 


-09- 


-1 OQA 


1160467 A 


24- 


-09- 


-1997 


0777946 A 


11- 


-06- 


-1997 


0853433 A 


15- 


-07- 


-1998 


10504944 T 


12- 


-05- 


-1998 


292020 A 


28- 


-10- 


-1998 


9606503 A 


29- 


-02- 


-1996 


5659613 A 


19- 


-08- 


-1997 


5668603 A 


16- 


-09- 


-1997. 



US 5739864 



14-04-1998 



AU 
AU 
BR 
CA 
CN 
EP 
EP 
JP 
NZ 
WO 
US 
US 



WO 9929114 



10-06-1999 



NONE 



Foim PCT/ISA/Z10 (patent family annex) (July 1992) 

4/27/05, EAST Version: 2.0.1.4 



